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ABSTRACT 

This paper describes the research and evaluation 
efforts of the Navy Substance Abuse Prevention Program at the 
University of Arizona. The importance and usefulness of formative and 
summative evaluations are discussed and specific evaluation tasks are 
identified. The first of these tasks is associated with personnel 
selection; the functions and qualifications of various personnel 
(site screeners, f acilitatorr, site coordinators, trainers, and 
students) are reviewed. The second task considered is personnel 
training. The training of screeners, facilitators, and site 
coordinators, and observation training are discussed. The third task 
described involves the intervention process. Viable sources of 
program mediation are discussed including facilitator 
characteristics, curriculum, classroom environment, and participant 
characteristics. The final evaluation task discussed is participant 
change. The primary goal of the program intervention is defined as 
change in drug and alcohol related behavior in the target population. 
The components described in this section focus on establishing the 
extent to which participants 1 knowledge, attitudes, and behaviors are 
alterod as a direct result of participation in the program. Five 
summative studies currently in progress are briefly outlined. 
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Navy Substance Abuse Prevention Program: 
Research and Evaluation 

The implicit goal of research and evaluation efforts 
entails generation and dissemination of continuous feedback 
regarding the strength and integrity of each program 
element, thus providing an empirically valid data source to 
complement strategic change, administrative decision 
making, problem solving, and ultimately, program evolution. 
The process oriented model for evaluation (see attached 
Figure) includes three formative (namely: personnel 
selection, personnel training, and process) and two 
summative (namely: participant change, and system impact) 
components which represent fourteen interrelated tasks. 

Inclusion of the formative components appears to be 
relatively novel within existing alcohol and drug 
prevention/intervention evaluation efforts* These three 
domains will enhance the probability of identifying "types" 
of curricular elements, "types" of learning environments, 
and "types" of participants that will contribute to 
successful intervention. Formative research and evaluation 
efforts will illuminate (and eventually be utilized to 
reduce and/or eliminate contamination) mediators of the 
knowledge, attitude, and behavioral change that the program 
is designed to accomplish. By examining process, the 
ability to determine the relative contribution of each 
component to the complete intervention package will be 
maximized. Continuous monitoring and refinement of each 
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intervention component should, ultimately, result in a 
"package" that is the best possible for the Navy 
population. The program will be flexible to the extent 
that it is effective. 

The Summative evaluation goals represent "anchors" for 
all formative components; i.e., Participant Change and 
System impact data will be utilized to dismantle the 
Formative components, separate effective from feckless 
pieces of the "package", strengthen efficacious content 
while simultaneously altering or replacing ineffective 
content, and finally, to demonstrate positive outcome to 
Navy, Civilian, and Scientific communities. 
Instrumentation for the Participant Change component 
includes assessment of knowledge, attitude, and behavior. 
System Impact is restricted to the Navy specified goals and 
objectives which focus upon positive change 4 .n terms of 
non-judicial punishment for alcohol and drug related 
incidents, work related accidents, sick days, hospital 
days, absenteeism, rate/rank reductions, and negative 
confrontation with either Naval or Civilian communities* 
Both Summative components are designed to assess impact, as 
well as providing a gauge for all Formative efforts. 

Collectively, Formative and summative Components 
combine to form a solid foundation for a systematic 
approach to research and evaluation which should 
effectively reduce many of the uncontrolled threats to 
validity that are quite obvious in any human service 
intervention. The vast majority of programs that resemble 



NAD SAP in terms of delivery have, more often than not, been 
plagued by a host of intervening variables which render 
interpretation of treatment effectiveness impossible. 
'Aside from obvious differences among program participants 
(e.g., demographic, social, psychological , physiological , 
and cognitive), human service interventions introduce 
intra- and inter-treatment variability as well. Potential 
threats to validity and integrity of human service 
interventions are overwhelming; but, standardization within 
and across intervention agents, measures, instructional 
techniques, and strategies can effectively reduce invalid 
interpretation and inference. 

Essential to the evolution of standardized 
intervention is a thorough understanding of all treatment 
components, their relative contribution to the desired 
outcome (in terms of the ratio of effort to quantity and 
quality of expected outcome), and reduction and/or 
elimination of extraneous sources of variability. Careful 
research design and thoughtful conceptual analysis are 
prerequisites for identification and comprehension of 
intervening variables. When factors that mediate treatment 
(both intra- and extra-treatment) are clearly understood, 
intervention effects can be traced throughout a network of 
outcome measures. Potential sources of variability 
inherent to The University of Arizona NADSAP efforts are 
restricted to the three Formative components. The specific 



evaluation tasks which constitute these general components 
are discussed in detail below. 

PERSONNEL SELECTION 
(1.1 through 1.6) 

Research and evaluation efforts associated with 
Personnel Selection are restricted to The University of 
Arizona employees: Trainers* Site Coordinators (Site level 
program managers), Screeners (Participant triage agents), 
and facilitators (intervention delivery agents). 
Eventually, collection of continuous data from all progr i 
personnel should result in a systematic reduction in 
uncontrolled sources of variability (thus eliminating many 
of the threats to internal and external val idity) by 
ensuring a high congruence between the program as intended 
and the program as delivered (integrity). Seduction in 
variability associated with each of the Personnel Selection 
components will occur as follows. 

1.1 - Screener Screening 

The primary function of Site Scteeners is 
preinter vent ion participant evaluation and triage. Hence, 
Screeners are expected (by way of a semistructured 
interview) to refer potential NADSAP participants to the 
most appropriate source of help; be it education (NAD SAP) , 
or counseling (CAAC), while simultaneously providing a 
positive, motivating introduction to the helping network. 



Since the vast majority of information which is utilized 
for assessment and subsequent referral involves self- 
report, the Screener position requires excellent 
communication skills (both verbal and nonverbal), as well 
as observation/inference skills to distinguish accurate or 
inaccurate behavioral accounts. Screeners are expected to 
track process throughout the interview, to be nonjudgmenial 
during the referral process, and to accurately articulate 
information garnered from a "record review" and the 
interview throughout the entire screening process. 

Site Screeners are selected by Site Coordinators on 
the basis of their skills in observation, tracking, 
communication, and decision making* Eventually, Screener 
screening will be contingent upon findings related to the 
Participant Change component! Referral accuracy will be 
estimated from the magnitude of Participant change, and a 
predictive model constructed from demographic, 
psychological, and skill characteristics of effective and 
ineffective on-1 ine screeners will be constructed to 
compliment the existing Screener selection process. 

1.2 - Pacilitator Screening 

Currently, site coordinators screen prospective 
facilitators prior to The university of Arizona Training. 
This screening process has consisted predominately of 
structured interviews which were developed at The 
University of Arizona. Although some improvement (as 
reflected by an increase in percentage pass rates in the 
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Facilitator training component) las been observed since 
implementation of this process. Variation across sites is 
indicative that the observed benefits are not consistent. 
The most obvious explanation for this inconsistency is that 
Site Coordinators represent a heterogeneous group that 
enter into the process with a variety of subjective 
impressions concerning the distinction between good 
facilitator prospects and inadequate ones. The facilitator 
screening package (a compilation of interview questions, 
paper and pencil instruments, and an "impressions 
inventory" for the Site Coordinator was created to reduce 
variability among training groups. Unfortunately, the 
degree of implementation during the facilitator screening 
process varies considerably across Site Coordinators. More 
often than not, pass rates (during the f aci lit at or 
training) are reflected in the extent that the facilitator 
screening package was employed during the screening 
process. Data indicate that the facilitator screening 
package can effectively reduce interindi vidual variability 
among perspective training groups. When, the package is 
used consistently by all Site Coordinators, facilitator 
variability within and across all NAD SAP sites will be 
minimized, thus enhancing the quality of The University of 
Arizona facilitator training. By allowing only individuals 
with observable skills into the training, training groups 
will decrease in size and pretraininq skill level will be 
amenable to more advanced training techniques. 
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Initial efforts to streamline the facilitator 
screening package in vol ved an extensive investigation of 
the predictive relationship between facilitator responses 
to Rokeach and Piro-B scales and their subsequent scores in 
The University of Arizona Facilitator Training. Although 
the fourteen subscales of these instruments accounted for 
statistically significant portions of variance in trainer 
assessments of skill levels, the practical significance of 
this endeavor was mediocre at best. Fortunately, these 
findings were interpreted as "promising", and use of the 
Rokeach and Firo-B scales was continued. 

Results from the initial effort forced recognition 
that a major problem associated with predicting trainer 
assessments of skill level from facilitator responses to 
the two instruments is related to the error increase due to 
an interaction between unacceptable intertrainer agreement 
on skill level and measurement error Inherent to the self- 
report information. Obviously, error originating from two 
(both in this instance) sources eliminates the possibility 
of accurately partitioning th> effects of each. 

Accordingly, a series of psychometric analyses were 
initiated to examine the construct validity of the Rokeach 
and Firo-B scales. A new data set was obtained, and factor 
analytic strategies were employed to examine the factorial 
validity of eacn instrument. The number of factors to be 
regained was forced to reflect the number of constructs 
purportedly measured by each instrument} eight for the 
Rokeach, and six for the Firo-B. Results from this 
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analysis (a principle components extraction technique, 
followed by orthogonal rotation) indicated that neither of 
the instruments yielded an interpretable factor structure. 
(If eigenvalues greater than one had been the criterion for 
factor retention, the Sokeach analysis would have resulted 
in sixteen factors, the Firo-B analysis resulted in 
thirteen factors.) Evidently* the Sokeach and Piro-B 
instruments fail to possess the properties which they have 
been attributed in the related literature* 

Failure to establish acceptable levels of construct 
validity within either of the predictive measures 
constituted the rationale for dismantling each scale, item 
by item, in an attempt to construct a new measure to 
successfully predict trainer assessments of facilitation 
skill level. The lack of factor- structure that was 
observed for both instruments provided convincing evidence 
that neither was reliable (and hence, by definition valid) 
for predicting facilitator success in the training. 

Individual items from both scales were entered 
separately into a regression analysis where trainer 
assessments of t he eight f aci 1 i tat ion sk i 1 1 s represented 
the dependent measure. The initial "sifting" process 
identified twelve items from the Sokeach, and twenty-two 
from the Firo-B that combined to explain between sixty-four 
and ninety-six percent of the variation in trainer 
evaluations of skill level. 
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To further refine the predictive accuracy of these 
items, another analysis was computed to eliminate overlap 
between all thirty-four items. This procedure eliminated 
another ten items (two from the Rokeach, and eight from the 
Piro-B). The remaining twenty-four items shared between 
seventy-eight and ninety-nine percent of the variance in 
the dependent measures. In essence, the sifting process 
identified twenty-four items from an original pool of 
ninety-four, that contribute significant amounts of 
predictive information to differentiate noor from good 
facilitation skills as perceived by The University of 
Arizona Trainers. This process is currently in a 
replication phase. 

1.3 - Site Coordinator Screening 

The Site Coordinator (site Level Program Manager) 
functions as middle-management; and hence, is responsible 
for ensuring that the program as conceived is congruent 
with the intervention that Navy participants receive. 
Responsibilities include Personnel Selection (office staff, 
Site Screeners, and Facilitators), facilitator and class 
scheduling, and liaison between local Navy personnel, local 
contract employees, and the central office in Tucson. 

Site Coordinators are selected by The University of 
Arizona management in accordance with their performance 
during a lengthy, semistructured interview (not drastically 
different fiom the interview that Site Coordinators use to 
select potential facilitators for the Training Component). 
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Ski 1 Is associated v 4 . th the Site Coordinator position 
include: Communication* Problem Solving* Decision Making* 
and Management. 

In an attempt to build a statistical model to 
discriminate "effective" from "ineffective" Site 
Coordinators* all of the on-line Site Coordinators 
responded to the Personal Orientation Inventory and the 
Strong-Campbell State-Trait Scale. To validate the utility 
of these instruments* the principal investigator* 
operations coordinator* and three trainers ranked each 
respondent according to their perceptions of performance in 
the role of Site Coordinator. Responses to the two 
instruments were utilized to predict rank. Similar to our 
experience with the facilitator screening package* results 
from this effort were unsuccessful. Future efforts will 
employ different measures to predict Site Coordinator 
performance. 

1*4 - Trainer Screening 

The screening process for trainers consists of a 
comprehensive interview with each program component 
(namely: Training* Operations* Curriculum* and 
Evaluation). The primary function of this interview 
process is to screen out applicants who do not have the 
skills in process training which are deemed essential to 
The University of Arizona philosophy. Applicants who 
successfully complete the interview process are hired (on a 
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probationary status) and given the opportunity to 
demonstrate their skills during two separate trainings with 
two different on-line trainers. This provides trainer 
applicants an opportunity to demonstrate their skills* and 
the on-the-job performance evaluation conducted by the co- 
trainers is the ultimate test to determine whether or not 
these applicants will be selected as active trainers. 

1.5 - Student Screening (Level 1) 

The goal of the screening process is tworold: the 
teferral of Naval Personnel to the most appropriate source 
of hely and the positive* motivating introduction of the 
individual into the helping network. In The University of 
Arizona program* this goal is achieved through the 
administrative screening process. 

The screening process involves a meeting between the 
Site Scr eener (cf. * 1.3) and the potential NADSAP 
participant. During this meeting* the Navy member 
completes the MAD IS intake form (Navy Alcohol and Drug 
Information System* which elicits information regarding 
demographic and service background* pre- and post-service 
legal history* and present and past drug and alcohol use. 
The Screener and Navy member review health and service 
records to identify and discuss relevant problems. The 
Screener's analysis of this information is then utilized to 
refer the Navy member to the most appropriate source of 
intervention. 
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The primary goal of evaluation and research into this 
particul ar component is to vnsute that al 1 referral 
decisions* are accurate, and hence, consistent across all 
screeners, i.e., that any given set of information will be 
interpreted similarily by the Screener, and subsequent 
referral decisions will reflect this interpret ition by 
demonstrating that any given individual will be referred to 
the same level of intervention regardless of who makes the 
referral. To this end, the research and evaluation 
component has created, and distributed a "pilot action 
matrix" to facilitate the screening process. This 
instrument focuses upon factors which are related to past 
and present alcohol and drug use, as well as the magnitude 
and direction of change evident in each of the indicators. 
The primary intent for this particular instrument is to 
reduce var iabil ity between screener's assessments and 
subsequent referral. By assigning only those who are 
likely to benefit from the MAD SAP experience into the 
course, personal success and program efficacy will increase 
substantially. 

1.6 - Student Screening (Level 2) 

The University of Arizona and Navy target individuals 
who are potentially amenable to an educational intervention 
only. Persons perceived as possessing drug nnd alcohol 
problems which extend beyond an educational intervention 
are sent, via the Level 1 screening process (1.5) to Level 
2 screening. This process is performed by Naval Personnel, 



and any treatment beyond the educational intervention 
provided by The University of Arizona is also a Navy 
function* Research and evaluation pertaining to Student 
Screening - Level 2 is not addressed by the current tasking 
order* Student Screening beyond Level 1 was incorporated 
into the evaluation model for the purpose of acknowledging 
the Navy's continuum of care* 

PERSONNEL TRAINING 

(2*1 through 2.4) 

Personnel Training is provided by The University of 
Arizona for Trainers, Site Coordinators, Screeners, and 
Facilitators* Trainees receive instruction in facilitation 
skills training and observational techniques associated 
with the skills which are strengthened during the 
facilitator training* Site Coordinators, Screeners, and 
Facilitators receive training which is tailored to their 
position* Evaluation and research concerning the training 
components focus upon enhancing inter-trainer agreement on 
facilitation skill level, establ ishing the generalizability 
of skills observed in the facilitator training to the 
classroom environment, and determining the effectiveness of 
each training component to overall program efficacy* A 
brief description of research and evaluation efforts 
associated with each of the training components follows* 



ERIC 



14 17 



2.1 - Screener Training 

All University of Arizonanti Screeners are required to 
attend the twenty -hour Paci 1 i tat or training which 
emphasizes eight facilitation skills: Tracking Content and 
Process, Empathy, Genuineness, Respect, Self-Disclosure, 
Openness to Feedback, Giving Objective Feedback, and Group 
Management. Since the primary function of the soreeners is 
participant assessment and triage*, skills which are deemed 
important for effective screening include communication 
(both verbal and non-verbal), observation/inference, and 
the ability to articulate large quantities of information 
in order to make the most appropriate referral decisions. 

Research and evaluation efforts related to this 
component focus primarily upon standardizing the referral 
process (by reducing subjective interpretation). In 
essence, the screener must choose one of two alternatives 
for each potential NADSAP participant! NAD SAP or CAAC 
(Level 2 screening). The "action matrix* discussed above, 
when used appropriately, virtually eliminates most 
sub jecti ve interpretation from the process, however, 
screeners do have the option of selecting the opposite 
choice provided that they provide in writing the rationale 
for deviation from the matrix. To date, research and 
evaluation regarding the Screener training component has 
been limited to matching screener decisions to "post- 
course" facilitator perceptions of "referral 
appropriateness", if facilitators perceive the referral as 
appropriate following their thirty-six hours of interaction 
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with the individual, the participant is returned to their 
command following NAD SAP* Should the facilitator feel that 
the participant needs addi ti onal help foil owing 
participation in NAD SAP, she/he can refer them to the CAAC 
for further screening and/or additional treatment. 

The ultimate goal of the administrative screening 
procedure is to make appropriate referrals during the 
initial intake interview. Future efforts* following the 
establishment of "Participant Characteristics" (cf.r 3.4) 
which are correlated with positive change attributable _o 
participation in NADSAP, involve incorporating these 
characteristics into the Screening process. Site Screeners 
will be trained to utilize all relevant information prior 
to the referral decision. Screening effectiveness will be 
established by calculating "hit rates" based upon all 
dependent measures included in the "Participant Change" 
component discussed below. 

2.2 - Pacilitator Training 

The University of Arizona Trainers who successfully 
complete the Trainer Screening (1.4) and Observation 
Training (2.4) meet prospective facil itators who have 
successfully completed the Facilitator screening Process 
(1.2) during The University of Arizona Facilitator Training 
(2.2). This intensive twenty-hour experience is designed 
to develop and refine the skills deemed necessary for 
_i.ective facilitation (namely: Tracking Content and 
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Process, Respect, Genuineness, Empathy, Openness to 
Feedback, Giving Objective Feedback, Group Management, and 
Self-Disclosure). During the Training sessions, 
prospective facilitators are given numerous opportunities 
to demonstrate and refine their skill level. Trainers 
observe process, evaluate individual skill level, model 
skills, and provide timely feedback to shape skills into 
acceptable levels. 

Evaluation of the Facilitator Training involves pre- 
and post-training assessment of facilitator perceptions 
regarding the importance of various characteristics and 
skills which are conducive to NAD SAP participant knowledge, 
attitude, and behavioral change. The self-report 
instrument is designed to elicit perceptions regarding the 
importance of twenty-two facilitator skills and 
characteristics in relation to effective facilitation, as 
well as self-assessment on each characteristic. Ratings of 
the importance of each item are construed as a measure of 
Ideal Facilitation correlates, and self-assessments are 
conceptualized as Real Facilitation Ability. The 
discrepancy between Real and Ideal perceptions is a 
projective measure of Self-Efficacy specific to each 
training participant. Hence, Training effectiveness is 
assessed from an Ideal perspective (i.e., How does 
participation in the Facilitator Training affect self- 
assessed facilitation ability?), and a Self-Efficacy 
perspective (i.e., How does participation in the 
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Facilitator Training affect perceptions regarding the 
belief that one is an effective facilitator?). 

Since all components of the MAD SAP evaluation model 
are inextricably interdependent, data emerging from the 
Facilitator Training module can eventually be employed to 
evaluate the effectiveness of each University of Arizona 
Trainer and each Facilitator Training according to the 
three domains described above. Opon completion of the 
Participant Change (4.1 through 4.3) components? 
facilitator effectiveness can be estimated from participant 
knowledge, attitude, and behavioral change, and 
subsequently matched to the facilitator's trainer(s). 
Estimates of trainer effectiveness will be inferred from 
facilitator effectiveness in the classroom. 

2*3 - site Coordinator Training 

The Site Coordinator (Site Level manager) represents 
the primary 1 ink between the Navy, The University of 
Arizona, and the MAD SAP employees at the Site level. Major 
responsibilities include scheduling classes and 
facilitators, managing the budget, screening facilitators 
and screeners, and organizing and managing mon thly 
facilitator meetings. 

Site Coordinator Training consists of three major 
components: An orientation at The University of Arizona 
following hire, periodical management assist visits, and an 
Annual Management conference. The orientation process 
lasts approximately one week and consists of an intense 
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overview of each program element. During orientation, new 
Site Coordinators meet with representatives from each 
program component (namely? Operations* Curriculum* 
Training, and Evaluation), and discuss issues, policy, and 
management strategies. The Annual Management conference 
consists of five days in Tucson, and two days in San Diego* 
During the first segment, all Site Coordinators meet to 
discuss issues, exchange ideas and management strategies* 
and participate in skills training in each area relevant to 
their position. The second segment merges all contract 
employees (Site Coordinators and Tucson staff) with Navy 
personnel from each Site. During this segment, military 
issues with contractors, and contractor issues with the 
military are discussed. 

2.4 - Observation Training 

The University of Arizona Trainers have the final 
input regarding who is, and who is not hired from the 
prospective facilitator pool. Hence, the importance of 
their skill evaluation ability cannot be overemphasized. 
Since the inception of this program, several observational 
rating forms have been developed and field tested to ensure 
that similar f aci litation skil 1 1 eve Is wil 1 receive 
identical assessments, regardless of which trainer 
evaluates them. Currently, an eight-item form is being 
used in the field. The eight behavior skills (namelyt 
Tracking Content and Process, Empathy, Respect, Self- 
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Disclosure, Openness to Feedback, Giving Objective 
Feedback* and Group Management) are evaluated separately 
for each of the facilitator trainees* and individual scores 
from these assessments determine who is able to continue 
the hiring process as a University of Arizona Facilitator. 

Initial efforts involved extracting demographic 
information from the completed employment forms retained in 
the Tucson Office, by combining this information with the 
evaluation scores assigned by The University of Arizona 
Trainers* we were able to provide feedback to the Trainers 
regarding the relationship between their skill assessments 
and the specified characteristics of age and education 
among the Facilitator pool. Although this information was 
interesting, as well as informative, it did not address 
agreement issues associated with the evaluation process* 
nor did it reflect the entire training population (files 
retained in the tucson office pertain to employees only. 
The sample for this investigation consisted of facilitators 
who had previously passed the Training which necessarily 
meant that they had received scores in excess of three 
across all eight skills). 

In order to better assess inter trainer agreement, 
independent trainer ratings were mandated (often times* two 
trainers conduct the facilitator training) to pave the way 
for more valid and comprehensive interpretations concerning 
the behavioral ratings. A brief questionnaire designed to 
elicit demographic information from all members of the 
training groups is also distributed during the facilitator 
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trainings. The instrument contains questions regarding 
respondents gender, age, ethnicity, marital status, NADSAP 
background, and self-reported experience with alcohol and 
drugs. 

By administering the questionnaire at the beginning of 
the training experience, demographic information is 
available on every person in the training, regardless of 
whether or not the training participant received parsing 
scores from the trainers. 

In a related effort, an analogue study was created to 
obtain independent trainer ratings of a video-taped segment 
of an actual training. Training tapes were edited, and a 
new tape depicting a single facilitatoi in a role play of 
the Group Management exercise was developed, the final 
version was approximately seven minutes in length. 

Each of The University of Arizona on-line Trainers was 
asked to view the tape, and to assign behavioral 
assessments to each of the eight facilitation skills 
presented on the eight itetti feedback form. Completion of 
this task represented the first time that each of the on- 
line Trainers had independently rated the same target 
person. From these data, estimates of inter-trainer 
agreement were established by computing multiple 
correlation coefficients for each trainer with all others 
for each of the facilitation skill assessments. These 
results were not particularly impressive; R ranged from a 
low of .396 to a high of .727. Apparently, the ambiguity 
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inherent to each of the eight facilitation skill categories 
induced the Trainers to define, and subsequently rate the 
skills according to theii own priorities conception of the 
specific behaviors that represent each of the eight skills. 

In an attempt to advance our understanding of the 
process underlying the evaluation technique, the original 
score matrix was inverted, and Trainers were clustered 
according to their skill assessment (Q-factor analysis). 
This procedure yielded three independent factors (or 
clusters) of trainers. Intertrainer agreement within 
factors ranged from R«.739 to .900 for factor one Trainers, 
and R-.704 to .808 for factor two Trainers. A single 
Trainer constituted factor three; within factor agreement 
was neither computable nor interpretable for factor three. 

Although intertrainer agreement within factors was 
acceptable, interfactor comparisons were generally weak. 
Multiple correlation coefficients calculated across trainer 
clusters (or factors) ranged from ft 8 *. 432 to .727. Since 
our ability to streamline an effective paper and pencil 
screening tool for facilitators (1.2) is inherently related 
to the extent that we can establish reliability within the 
criterion measure (Viz., Trainer ratings of facilitation 
skill level), the results of the analogue study are a step 
toward this end. 

Future efforts involve continued collection of 
information regarding Facilitator Training participant 
demographic characteristics from every training 
participant, regardless of their performance. This 
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information, in conjunction with the independent trainer 
evaluations of participant performance will yield 
sufficient data to provide trainers with practical feedback 
concerning potential biases which are integrated into their 
behavioral assessment. This feedback, combined with 
independent Trainer ratings of the edited training tapes 
(collected in a controlled environment) cap then be 
implemented to enhance intertrainer reliability. Immediate 
and persistent feedback should ultimately result in a 
reduction of intertrainer variability across the behavioral 
ratings. Furthermore, the facilitator demographic 
information will be employed to identify facilitator 
characteristics associated with (and predictive of) success 
in the Facilitator Training. 

As noted earlier, the Personnel Training component of 
the NADSAP evaluation model focuses primarily on reducing 
and/or eliminating uncontrolled variables inherent to the 
overall intervention effort. This brief description of the 
Personnel Training evaluation tasks has highlighted several 
potential sources of variability, identified possible 
strategies to reduce this variabili ty, and pros nted 
previous and proposed methods of decreasing these potential 
threats to the integrity of the entire intervention effort. 
Mo doubt, much more effort is required to achieve this 
goal. The Hodel for Evaluation of NADSAP provides insight 
for increased understanding by providing direction for 
future efforts. 
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PROCESS 

(3.1 through 3*4) 

While it is recognized that standardization is 
necessary for evaluation of human service intervention 
programs* it is also evident that standardization is not 
sufficient for program success* Hence, the rationale for a 
process component within the evaluation effort* Even 
though evaluation efforts directed toward Personnel 
Selection and Personnel Training are reducing variability 
among numerous factors, not all mediating variables can be 
eliminated* A thorough understanding of the intervention 
process can enhance the effectiveness of NADSAP. As shown 
in the Evaluation Model , four distinct factors with 
mediational potential have been incorporated into the 
Process component* Obviously* Facilitator Characteristics, 
Curriculum, Classroom Environment, and Participant 
Characteristics are Classroom Environment, and Participant 
Characteristics are viable sources of program mediation* 

The primary impetus of the Process evaluation 
component involves examination of mediating factors within 
the intervention itself* Upon establishing the effects of 
each factor, as well as the interaction between these 
factors* this knowledge can be employed to enhance 
intervention outcome* For example, in a previous 
investigation* age of participant interacted with 
facilitator gender to affect participant course and 
facilitator evaluations* Younger participants possessed 
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more positive views of courses facilitated by female 
facilitators. Conversely, as participant age increased, 
evaluations of female facilitated MAO SAP classes decreased. 
Should this phenomena emerge repeatedly in future 
investigations, it would behoove program administrators to 
mandate female facilitators for classes composed of young 
Participants, and male facilitators for classes that are 
composed of older (30+) participants. Ideally, a thorough 
understanding of all variables that mediate intervention 
effectiveness would guide manipulation of these factors to 
maximize desired change in the outcome measures. The four 
categories included in the Process component of evaluation 
are; Facilitator Characteristics, Curriculum, Classroom 
Environment, and Participant Characteristics. 



3.1 - Facilitator Characteristics 

Aside from the Trainer evaluations of skill level 
during the Facilitator Training, the only feedback that 
facilitators receive is provided by the NADSAP participants 
following the course. Participant perceptions of 
fac' 1 *utor and course effectiveness are important to 
evaluation because they are indicative of facilitator 
competency as perceived by the target population. 
Facilitator Characteristics (demographic, cognitive, 
psychological, and physiological) are probable mediators of 
participant perceptions of competency. Examination of 
Facilitator Characteristics, in conjunction with 
Participant Change, will strengthen the Facilitator 



Screening (1.2) and Facilitator Training (2.2) components 
of the evaluation effort. Furthermore, insights for 
molding the NADSAP efforts around the intervention 
population to maximize knowledge, attitude, and behavioral 
change will evolve through investigation of process. 

Measurement of knowledge, attitude, and behavior, both 
pre- and post-NADSAP, reflect course effectiveness; the 
NADSAP facilitator and course evaluations reflect 
participant perceptions of course effectiveness. The 
difference between participant change and perceptions of 
usefulness is of interest because Facilitator 
Characteristics potentially mediate both; i.e., some 
Facilitator Characteristics mediate actual change, and 
others mediate perceived change. 

3.2 - Curriculum 

The curriculum, or plan of presentation, is composed 
of several experiential and instructional components, each 
accompanied by specific goals and objectives, further, 
each of the modules can be presented in a variety of 
sequences. Aside from examining the relative effectiveness 
of each module, the ordering and the sequencing of 
presentation are likely mediators of Participant Change. 
Comprehensive experimental designs will be developed to 
examine the effectiveness of each curricular component, as 
well as to establish the most effective order and sequence 
of presentation. if some modules arc found to be 
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nonef f icacious according to the desired knowledge, 
attitude, and behavior change, they will either be removed 
or strengthened to contribute to the overall intervention. 
Like wis e, by manipul a ting order and sequence of 
presentation, we can determine which combination yields 
maximum change in the outcome measures. 



3,3 - Classroom Environment 

The educational literature is literally bloated with 
studies which examine the effects of environmental 
contingencies and the learning process. Educators and 
psychologists have documented the impact of a host of 
environmental variables on subsequent learning outcomes. 
Factors such as class size, day versus night scheduling, 
location, length of sessions, and class "character" are 
likely mediators of the intervention process. The 
Classroom Environment component of the evaluation model was 
included to examine these variables. Similar to the 
proposed methodol ogies described above, experimental 
designs can be utilized to investigate the effect of 
variations in learning environments. This information, 
will be utilized to maximize intervention effectiveness. 



3.4 - Participant Characteristics 

Although Participant Characteristics such as age, 
rank, educational level, length of time in the service, 
gender, family background, pay rate, previous drug and 
alcohol experience, "quality of life", and ethnicity are 
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not likely to be affected by participation in NADSAP, 
individual differences and intervention outcome can be 
combined to maximize participant change* By examining 
Participant Characteristics and knowledge, attitude, and 
behavior change following NADSAP, the mediational effects 
of intra- and inter-individual differences can be harnessed 
and controlled to maximize intervention effectiveness. 
Previous studies which have focused upon participant 
characteristics and participant perceptions of "course 
usefulness" have demonstrated that an interaction does 
exist. Our goals and objectives related to this component 
are inherently related to the general izabi 1 ity of these 
interactions to actual change in knowledge, attitude, and 
behavior. 

Ideally, repeated examination of the relationship 
between process variables inherent to the NADSAP 
intervention should provide a weal th of information 
pertaining to Facilitator Characteristics, Curriculum 
Modules, Classroom Environments, and Participant 
Characteristics which are conducive to successful 
intervention. Moreover, insight concerning the interaction 
between all process variables can be useful in "molding" 
the entire intervention package to the target population to 
ensure maximum change in outcome. 
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PARTICIPANT CHANGE 
(4.1 through 4.3) 

The primary goal of The University of Arizona NADSAP 
intervention is to change drug and alcohol related behavior 
in the target population. While knowledge is necessary for 
behavioral change, it alone is not sufficient. When the 
Process components (3.1 through 3.4) are effectively 
manipulated to ensure intervention effectiveness, 
participant attitudes should change also. The Participant 
Change components focus upon establishing the extent to 
which participant knowledge, attitude, and behavior are 
altered as a direct result of participation in NADSAP. 
Attitudes of interest include attitudes toward alcohol and 
drug use, attitudes toward work, and most importantly, 
attitudes toward self. The University of Arizona 
philosophy is geared toward increasing self -awareness and 
responsibility toward self and others. The Participant 
Change component of the evaluation model was included to 
investigate the extent that the intervention accomplishes 
these goals. 



4*1 - Knowledge Change 

Knowledge is assessed with the forty-it em 
questionnaire developed by Halfetti (1971). Information 
pertaining to scale construction, validation and related 
psychometric properties, and norms can be obtained from The 
University of Arizona. 
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4*2 - Attitude Change 

The instrument for assessing attitude change consists 
of a composite of previously developed scales which have 
demonstrated acceptable reliability and validity, as well 
as a documented relationship to either alcohol and drug 
use r or alcohol and drug use/abuse prevention/intervention* 
Specifically, modifications of the Rosenbaum (1980) Self- 
Control Scale, Reynold's (1982) version of the Marlow- 
Crowne (1960) Social Desirability Scale, Zuckerman's (1979) 
Sensation Seeking scale (subscales include Disinhibition, 
Susceptibility to Boredom, Novel Experience Seeking, and 
Thrill and Adventure Seeking), the Rathus (1973) 
Assertiveness Inventory, Rotter's (1966) Locus of control 
measure, the CES-D Depression scale (Radloff, U 7), Adam's 
measure of Ego-Identity (1983), and Rosenberg's (1965) 
Self -Esteem Scale. 

4*3 - Behavior Change 

Actual behavioral measures are extracted from the Navy 
Alcohol and Drug Information System (NADIS) and the 
participant's medical and personnel records during the 
screening (1*5). These data include BAC at the time of 
arrest (if applicable), pre-service arrests, convictions 
and waivers, court martials, reductions in pay, rate, and 
rank, alcoho 1 and drug related incidents (traffic and 
otherwise), as well as several self-report measures aimed 
at eliciting extent of alcohol and drug use. 
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In addition* two measures of drinking quantity and 
frequency are included to examine intervention 
effectiveness as mediated by the extent of drinking 
problems. These measures are the Alcoholism Indicator 
Scale (Halfetti & Simon* 1974) and the HAST. 

5.1 - Formative Evaluation 

Formative evaluation represents findings from each of 
the components which constitute the Personnel Selection* 
Personnel Training* and Process evaluation modules. 
Information obtained from tasks 1.1 through and including 
3.4 will be "fed back" to all other components. 

5.2 - Sumnatlve Evaluation 

Summative evaluation represents findings from each of 
the components which constitute the Participant Change 
evaluation module. Information obtained from tasks 4.1 
through and including 4.3 will be "fed back" to the Navy* 
Civilian* and Scientific communities. 

SUMMATIVE STUDIES-in progress 

FCP/NADSAP-DUl/DWI/OUl: two year longitudinal study 

currently implemented ati 
San Diego 
Bremerton 

Data collection: record reviews and questionnaires. 

Measurement intervals: Pre/Post* 3*6*9*12*18* & 24 

month followup. 

Sample Size to date: 241. 
Anticipated Mi 700. 
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Supervisor Pollowupl: post test only, currently implemented 

at: 

Great takes 

Data Collection: questionnaire. 

Measurement intervals: 3-month followup. 

Sample size to data: 143. 
Anticipated N: -ongoing 



Supervisor Pollowup2: pre/post, currently implemented at: 

San Diego 
Bremerton 

Data Collection: questionnaire. 

Measurement intervals: Pre-NADSAP, and 3-month 

followup. 

Sample size to date: 46 
Anticipated N: 60 

Participant Followup: on year* Cross-sectional study 

currently implemented at: 
San Diego 
Bremerton 
Pearl Harbor 
Great takes 
Jacksonville 
Yokosuka 

Data collection: NADIS, NADSAP Client intake form, & 

Questionn .ire. 

Measurement intervals: NAD IS & intake data-pre NADSAP; 

questionnaires-3, 6, 9, & 12 
months following class 
completion. 

Sample size to date: 119 
Anticipated N: 1800 



Student Knowledge Change: Pre and Post course implemented 

at: 

Various Sites (n«16) 

Data Collection: questionnaire. 

Measurement interval: Pre and Post NADSAP. 

Sample size to date: 3126. 
Anticipated N: ongoing. 
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